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WHAT IS CLAIMED IS: 
(TP An image processing method for describing a real 
image as a virtual object using space data formed based 
on a photo image of the real object, and generating a 
5 virtual image of the virtual object using the space 
data, comprising the steps of: 

capturing the photo image of the real object from 
a real camera viewpoint position while illuminating the 
real object by an illumination light source placed at a 
10 real illumination position; 

converting the photo image into space data in the 
form of an object; and 

storing the space data in a memory together with 
various illumination conditions at the real 
15 illumination position to allow a search at a later time. 

2. The method according to claim 1, wherein the 
illumination position of the illumination light source 
is set at a plurality of different positions, and 
position information and an illumination condition at 

2 0 each illumination position are stored in the memory 
together with corresponding space data. 

3. The method according to claim 1, wherein the 
camera viewpoint position is set at a plurality of 
different positions, and space data obtained at each 

2 5 viewpoint position is stored together with the 
viewpoint position information. 
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4. The method according to claim 1, wherein the 
space data is ray space theory data. 

5. The method according to claim 1, wherein the 
illumination condition is varied in a plurality of ways 

5 per illumination position, and space data corresponding 
to the individual illumination conditions are stored in 
the memory. 

CI 6 . An image processing method for describing a real 

91 image as a virtual object using space data formed based 

Hi 10 on a photo image of the real object, and generating a 

01 

iyj virtual image of the virtual object using the space 

3 data, comprising the steps of: 

ffj storing space data generated from a photo image 

B 9 

together with a real camera viewpoint position, and an 
15 illumination position and illumination condition of a 
real illumination light source; 

generating coordinates of a virtual image of an 
object at a user viewpoint position on the basis of the 
real camera viewpoint position and the space data; and 
20 correcting pixel values of the virtual image on 

the basis of the real illumination position and real 
illumination condition of the real illumination light 
source, and a virtual illumination position and virtual 
illumination condition set for a virtual illumination 
2 5 1 ight source . 
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7. The method according to claim 6, wherein the 
illumination position of the real illumination light 
source is set at a plurality of different positions, 
and position information and an illumination condition 

5 at each illumination position are stored in the memory 
together with corresponding space data. 

8. The method according to claim 6, wherein the real 
camera viewpoint position is set at a plurality of 

f l different positions, and space data obtained at each 

H- J 10 viewpoint position is stored together with the 

■a? 5 

t H J viewpoint position information. 

a 9. The method according to claim 6, wherein the 

pjj space data is ray space theory data. 

yi 10. The method according to claim 6, further 

r| 15 comprising the step of changing and setting an 

illumination condition of the virtual illumination 
light source to be an arbitrary value, and wherein the 
correction step includes the step of correcting the 
pixel values in accordance with the changed 
20 illumination condition. 

11. The method according to claim 6, wherein when a 
plurality of virtual light sources are ON at the same 
time, a plurality of correction results which are 
corrected in accordance with the respective virtual 
2 5 illumination light sources are added for one pixel 
position to obtain a final pixel value. 
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12. The method according to claim 6, wherein when 
some of a plurality of virtual light sources are ON at 
the same time, a plurality of correction results which 
are corrected in accordance with the respective ON 

5 virtual illumination light sources are added for one 
pixel position to obtain a final pixel value. 

13. The method according to claim 6, further 
comprising the steps of: 

m determining if the corrected pixel value 

fli 10 saturates; and 

u] virtually changing illumination conditions of 

~ some or all of the virtual light sources in a direction 

3= to decrease luminance or illuminance of an illumination 

lit when the pixel value saturates, and 

15 wherein the correction step is executed again in 

accordance with the changed illumination condition. 

14. The method according to claim 6, wherein the 
illumination condition includes information indicating 
whether or not an illumination is turned on. 

20 15. The method according to claim 6, wherein the 
illumination condition includes illuminance or 
luminance . 

16. The method according to claim 6, wherein the 
illumination condition includes a relative position 
25 between the virtual light source and object. 
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17. The method according to claim 6, wherein the 
illumination condition includes color of illumination. 

18. An image processing apparatus for describing a 
real image as a virtual object using space data formed 

5 based on a photo image of the real object, and 

generating a virtual image of the virtual object using 
the space data, comprising: 

means for capturing the photo image of the real 
object from a real camera viewpoint position while 
10 illuminating the real object by an illumination light 
source placed at a real illumination position; 

conversion means for converting the photo image 
into space data in the form of an object for each 
pixel; and 

15 means for storing the space data in a memory 

together with various illumination conditions at the 
real illumination position to allow a search at a later 
time . 

19. The apparatus according to claim 18, wherein the 
20 illumination position of the illumination light source 

is set at a plurality of different positions, and 
position information and an illumination condition at 
each illumination position are stored in the memory 
together with corresponding space data. 
25 20. The apparatus according to claim 18, wherein the 
camera viewpoint position is set at a plurality of 
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different positions, and space data obtained at each 
viewpoint position is stored together with the 
viewpoint position information. 

21. The apparatus according to claim 18, wherein the 
5 space data is ray space theory data. 

22. The apparatus according to claim 18, wherein the 
illumination condition is varied in a plurality of ways 

*f per illumination position, and space data corresponding 

a:; 

to the individual illumination conditions are stored in 
8*1 J 10 the memory. 

m 

23. An image processing apparatus for describing a 

a real image as a virtual object using space data formed 



based on a photo image of the real object, and 
generating a virtual image of the virtual object using 

15 the space data, comprising: 

means for storing space data in the form of an 
object for each pixel of a photo image together with a 
real camera viewpoint position, and an illumination 
position and illumination condition of a real 

20 illumination light source; 

generation means for generating coordinates of a 
virtual image of an object at a user viewpoint position 
on the basis of the real camera viewpoint position and 
the space data ; and 

25 correction means for correcting pixel values of 

the virtual image on the basis of the real illumination 
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position and real illumination condition of the real 
illumination' light source, and a virtual illumination 
position and virtual illumination condition set for a 
virtual illumination light source. 

24. The apparatus according to claim 23, wherein the 
illumination position of the real illumination light 
source is set at a plurality of different positions, 
and position information and an illumination condition 
at each illumination position are stored in the memory 



L'; 10 together with corresponding space data 



25. The apparatus according to claim 23, wherein the 
real camera viewpoint position is set at a plurality of 
different positions, and space data obtained at each 

viewpoint position is stored together with the 

O 

CI 15 viewpoint position information. 

26. The apparatus according to claim 23, wherein the 
space data is ray space theory data. 

27. The apparatus according to claim 23, further 
comprising means for changing and setting an 

20 illumination condition of the virtual illumination 

light source to be arbitrary values, and wherein said 
correction means corrects the pixel values in 
accordance with the changed illumination condition. 

28. The apparatus according to claim 23, wherein when 
25 a plurality of virtual light sources are ON at the same 

time, a plurality of correction results which are 
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corrected in accordance with the respective virtual 
illumination* light sources are added for one pixel 
position to obtain a final pixel value. 

29. The apparatus according to claim 23, wherein when 
5 some of a plurality of virtual light sources are ON at 
the same time, a plurality of correction results which 
are corrected in accordance with the respective ON 
virtual illumination light sources are added for one 

f* pixel position to obtain a final pixel value. 

fif 10 30. The apparatus according to claim 23, further 

comprising: 

gj 

a means for determining if the corrected pixel 

flj value saturates; and 

U| means for virtually changing illumination 

•SB* 

jp 15 conditions of some or all of the virtual light sources 

in a direction to decrease luminance or illuminance of 
an illumination when the pixel value saturates, and 

wherein said correction means is executed again 
in accordance with the changed illumination condition. 

20 31. The apparatus according to claim 23, wherein the 
illumination condition includes information indicating 
whether or not an illumination is turned on. 
32. The apparatus according to claim 23, wherein the 
illumination condition includes illuminance or 

2 5 luminance. 
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33. The apparatus according to claim 23, wherein the 
illumination condition includes a relative position 
between the virtual light source and object. 

34. The apparatus according to claim 23, wherein the 
5 illumination condition includes color of illumination. 

35. An image processing method for generating a 
shadow image of a virtual object itself in a virtual 
space, comprising the steps of: 

sensing an image of a real object by a camera 
10 while illuminating the real object corresponding to the 
virtual object from a light source at a predetermined 
illumination position; 

extracting a silhouette of the real object from 
the image of the real object; 
£5 15 adding preferred image information as a shadow to 

pixels in the silhouette; and 

storing the silhouette image together with 
position information of the light source, the 
silhouette image being able to be found by search and 
20 retrieved at a later time. 

36. The method according to claim 35, wherein the 
preferred image information includes a predetermined 
black pixel value and a predetermined transparency 
value . 

25 37. The method according to claim 35, wherein the 
silhouette image undergoes a blur process. 



y 5 
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38. The method according to claim 37, wherein the 
degree of blur is changed in correspondence with 
distance from the virtual object to a virtual light 
source . 

39. The method according to claim 35, wherein a 
viewpoint position of the camera matches the position 
of the illumination light source. 

40. An image processing method for generating a 
shadow image of a virtual object itself in a virtual 
space, comprising the steps of: 

storing space data of the virtual object, a 
shadow image of a real object corresponding to the 
virtual object, and a position of an illumination light 
source upon forming the shadow image in a predetermined 
memory; 

reading out the shadow image from the memory in 
accordance with a position of a virtual illumination, 
and a relative position of the virtual object; and 

mapping the readout shadow image on a 
predetermined mapping plane. 

41. The method according to claim 40, wherein the 
mapping plane is determined on the basis of a bounding 
box of the virtual object. 

42. The method according to claim 40, further 
comprising the step of changing and setting an 
illumination condition of the virtual illumination 
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light source to be an arbitrary value, and wherein the 
correction step includes the step of correcting the 
pixel values in accordance with the changed 
illumination condition . 
5 43. The method according to claim 40, wherein when a 
plurality of virtual light sources are ON at the same 
time, a plurality of correction results which are 

ri 

*i corrected in accordance with the respective virtual 

i § % 

J* 3 illumination light sources are added for one pixel 

'2: 10 position to obtain a final pixel value. 

Pi! 

JSf 44. The method according to claim 40, wherein when 

% some of a plurality of virtual light sources are ON at 

Mi the same time, a plurality of correction results which 

0! are corrected in accordance with the respective ON 

CI 

C! 15 virtual illumination light sources are added for one 

pixel position to obtain a final pixel value. 
45. The method according to claim 40, wherein when 
the illumination is located at a high-level position, a 
mapping plane with a simple shape is generated. 

20 46. An image processing apparatus for generating a 
shadow image of a virtual object itself in a virtual 
space, comprising: 

means for sensing an image of a real object by a 
camera while illuminating the real object corresponding 

25 to the virtual object from a light source at a 
predetermined illumination position; 
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means for extracting a silhouette of the real 
object from the image of the real object; 

means for adding preferred image information as a 
shadow to pixels in the silhouette; and 

means for storing the edge image together with 
position information of the light source, the 
silhouette image being able to be found by search and 
retrieved at a later time. 

47. The apparatus according to claim 46, wherein the 
preferred image information includes a predetermined 
black pixel value and a predetermined transparency 
value . 

48. The apparatus according to claim 46, wherein when 
the viewpoint position of the camera is different from 
the position of the light source, the silhouette image 
is corrected by re-projecting the silhouette image to 
have the position of the light source as a viewpoint 
position . 

49. The apparatus according to claim 46, further 
comprising means for performing a blur process of the 
silhouette image. 

50. The apparatus according to claim 49, wherein the 
degree of blur is changed in correspondence with 
distance from the virtual object to a virtual light 
source . 
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51. The apparatus according to claim 46, wherein a 
viewpoint position of the camera matches the position 
of the illumination light source. 

52. An image processing apparatus for generating a 
5 shadow image of a virtual object itself in a virtual 

space, comprising the steps of: 

means for storing space data of the virtual 
El object, a shadow image of a real object corresponding 

fjl to the virtual object, and a position of an 

111 10 illumination light source upon forming the shadow image 

yj in a predetermined memory; 

a means for reading out the shadow image from the 

rjj memory in accordance with a position of a virtual 

III illumination, and a relative position of the virtual 

5J 15 object; and 

means for mapping the readout shadow image on a 
predetermined mapping plane . 

53. The apparatus according to claim 52, wherein the 
mapping plane is determined on the basis of a bounding 

20 box of the virtual object. 

54. The apparatus according to claim 52, further 
comprising means for changing and setting an 
illumination condition of the virtual illumination 
light source to be an arbitrary value, and wherein said 

2 5 correction means corrects the pixel values in 

accordance with the changed illumination condition. 
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55. The apparatus according to claim 52, wherein when 
a plurality of virtual light sources are ON at the same 
time, a plurality of correction results which are 
corrected in accordance with the respective virtual 

5 illumination light sources are added for one pixel 
position to obtain a final pixel value. 

56. The apparatus according to claim 52, wherein when 
some of a plurality of virtual light sources are ON at 
the same time, a plurality of correction results which 

10 are corrected in accordance with the respective ON 

virtual illumination light sources are added for one 
pixel position to obtain a final pixel value. 

57. The apparatus according to claim 52, wherein when 
the illumination is located at a high-level position, a 

15 mapping plane with a simple shape is generated. 

58. An image processing method for generating a 
shadow image of a virtual object itself in a virtual 
space, comprising the steps of: 

extracting a silhouette of the virtual object 
20 viewed from a predetermined illumination position on 
the basis of space data which expresses the virtual 
object; 

adding preferred image information as a shadow to 
pixels in the silhouette; and 
25 mapping the silhouette image on a predetermined 

mapping plane. 
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59. An image processing apparatus for generating a 
shadow image of a virtual object itself in a virtual 
space, comprising: 

means for extracting a silhouette of the virtual 
5 object viewed from a predetermined illumination 

position on the basis of space data which expresses the 
virtual object; 

C= means for adding preferred image information as a 

yl shadow to pixels in the silhouette; and 

Oj 10 means for mapping the silhouette image on a 

& 

y predetermined mapping plane . 

g 60. An image processing apparatus for mixing a 

il virtual space with a real space illuminated by a real 

illumination device, and presenting the mixed space, 
15 comprising: 

change means for changing an illumination 
condition of the illumination device; 

first setting means for setting an illumination 
program parameter value corresponding to a value of the 
20 changed illumination condition; 

rendering means for rendering a virtual image 
using a predetermined illumination condition program 
parameter, and mixing the virtual image with the real 
space; and 

25 second setting means for setting, in a rendering 

routine of the virtual image of said rendering means, 
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the illumination condition program parameter set by- 
said settings means in correspondence with the 
illumination condition changed by said change means. 

61. The apparatus according to claim 60, wherein the 
illumination condition includes one of illuminance, 
color, and an illumination direction. 

62. The apparatus according to claim 60, wherein said 
change means comprises GUI means operated by a user to 
display control buttons for graphically changing 
illuminance, color, and an illumination direction of 
the illumination device on a display screen. 

63. The apparatus according to claim 60, wherein said 
change means comprises GUI means operated by a user, 
and said GUI means displays an image that simplifies 
the illumination device and allows the user to instruct 
to change a given illumination condition by designating 
a portion to be adjusted in the image using a pointing 
device . 

64. The apparatus according to claim 60, wherein said 
first setting means includes means for reading out a 
parameter pre-stored in a predetermined memory. 

65. An image processing method for mixing a virtual 
space with a real space illuminated by a real 
illumination device, comprising: 

the step of changing an illumination condition of 
the illumination device; 



the first setting step of setting an illumination 
program parameter value corresponding to a value of the 
changed illumination condition; 

the step of rendering a virtual image using a 
5 predetermined illumination condition program parameter, 
and mixing the virtual image with the real space; and 

the second setting step of setting, in a 
rendering routine of the virtual image, the 
illumination condition program parameter set in 
^ 10 correspondence with the illumination condition changed 

E'll 

Ol in the change step. 

y i 

CH 66. The method according to claim 65, wherein the 

a 

£1 first setting step includes the step of reading out a 

Six 

fy parameter pre-stored in a predetermined memory. 

f| 15 67. The method according to claim 65, wherein the 

change step includes the step of displaying a GUI 
operated by a user to display control buttons for 
graphically changing illuminance, color, and an 
illumination direction of the illumination device on a 
20 display screen. 

68. The method according to claim 65, wherein the 
change step includes the step of displaying a GUI 
operated by a user, and the GUI displays an image that 
simplifies the illumination device and allows the user 
25 to instruct to change a given illumination condition by 
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designating a portion to be adjusted in the image using 
a pointing device. 

69. A storage medium that stores a control program 
for making a computer execute an image process for 
describing a real image as a virtual object using space 
data formed based on a photo image of the real object, 
and generating a virtual image of the virtual object 
using the space data, said control program including: 

a code of the capture step of capturing the photo 
image of the real object from a real camera viewpoint 
position while illuminating the real object by an 
illumination light source placed at a real illumination 
position; 

a code of the conversion step of converting the 
photo image into space data in the form of an object; 
and 

a code of the storage step of storing the space 
data in a memory to allow a search together with 
various illumination conditions at the real 
illumination position . 

70. A storage medium that stores a control program 
for making a computer execute an image process for 
describing a real image as a virtual object using space 
data formed based on a photo image of the real object, 
and generating a virtual image of the virtual object 
using the space data, said control program including: 



* • 



a code of the storage step of storing space data 
generated from a photo image together with a real 
camera viewpoint position, and an illumination position 
and illumination condition of a real illumination light 
5 source; 

a code of the generation step of generating 
coordinates of a virtual image of an object at a user 
£| viewpoint position on the basis of the real camera 

itJl viewpoint position and the space data; and 

jffj 10 a code of the correction step of correcting pixel 

jrjj values of the virtual image on the basis of the real 

* illumination position and real illumination condition 

Si of the real illumination light source, and a virtual 

ill illumination position and virtual illumination 

£-5 s 

.3: i 

15 condition set for a virtual illumination light source. 
71. A storage medium that stores a control program 
for making a computer execute an image process for 
generating a shadow image of a virtual object itself in 
a virtual space, said control program including: 

2 0 a code of the image sensing step of sensing an 

image of a real object by a camera while illuminating 
the real object corresponding to the virtual object 
from a light source at a predetermined illumination 
position ; 
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a code of the extraction step of extracting a 
silhouette of the real object from the image of the 
real object; 

a code of the addition step of adding preferred 
5 image information as a shadow to pixels in the 
silhouette; and 

a code of the storage step of storing the 
silhouette image together with position information of 
the light source, the silhouette image being able to be 
FU 10 found by search and retrieved at a later time. 

hi 72 . A storage medium that stores a control program 

01 

3 for making a computer execute an image. process for 

m generating a shadow image of a virtual object itself in 

II a virtual space, said control program including: 

l{ 15 a code of the storage step of storing space data 

of the virtual object, a shadow image of a real object 
corresponding to the virtual object, and a position of 
an. illumination light source upon forming the shadow 
image in a predetermined memory; 
2 0 a code of the read-out step of reading out the 

shadow image from the memory in accordance with a 
position of a virtual illumination, and a relative 
position of the virtual object; and 

a code of the mapping step of mapping the readout 
2 5 shadow image on a predetermined mapping plane. 



- 56 - 



f 




73. A storage medium that stores a control program 
for making a> computer execute an image process for 
generating a shadow image of a virtual object itself in 
a virtual space, said control program including: 

5 a code of the extraction step of extracting a 

silhouette of the virtual object viewed from a 
predetermined illumination position on the basis of 
space data which expresses the virtual object; 

a code of the addition step of adding preferred 
10 image information as a shadow to pixels in the 
silhouette; and 

a code of the mapping step of mapping the 
silhouette image on a predetermined mapping plane. 

74. A storage medium that stores a control program 
15 for making a computer execute an image process for 

generating a shadow image of a virtual object itself in 

a virtual space, said control program including: 

a code of the step of changing an illumination 

condition of the illumination device; 
20 a code of the first setting step of setting an 

illumination program parameter value corresponding to a 

value of the changed illumination condition; 

a code of the step of rendering a virtual image 

using a predetermined illumination condition program 
2 5 parameter, and mixing the virtual image with the real 

space; and 
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a code of the second setting step of setting, in 
a rendering routine of the virtual image, the 
illumination condition program parameter set in 
correspondence with the illumination condition changed 
in the change step. 
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